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ABSIKACI 

The advantages of tie advarsarj ©valuation model are 
argued* Adversary evaluation utilises a Judicial, argumentative 
approach and promotes the use of multiple types and sources of data* 
It grants an investigative /license to dig out the data to a greater 
extent than more traditional approaches* Some of the payoffs of the 
adversary approach arei (1) the critical input of colleagues is 
considered at the most opportune time-^^prior to actual investigation; 
(2) the inclusion of provocative questions which address the real 
issues of various subgroups is encouraged, resulting in the selection 
or development of higher guality instruments; (3) the model 
explicitly encourages the use of qualitative as well as quantitative 
information; (4) it grants the fr.eedom of investigative license; (5) 
because polar positions are represented, people feel that issues they 
believe to be important are not overlooked; and {6) the model 
provides a built in meta-evaluation* (EC) 
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2« PAYOFFS OF ADVERSARY EVALUATION^ 

N4TieNAL iNSTiTUTE OP ' 

TMis DQcuMiNT N&s BiEN RipeQ. Northwest R^glonal EduGatlonal L^oratory 

BUCED ijt&CTLy AS HECEivEO FROM 
THi PERSON □» OBGfeNlZATiONO^lGiN- 

ATtN& IT POINTS viivv OH OPINIONS THnHiA^ P q;ar»h^^ 

i7*.TeD DO NOT NEcei$ARiLY REPRi- Anuiu^s ^ ^ s^cnss 

lENVOFFlCiftU NATiON&L INSTSTUTE OF 
EDUCATION POSITION OR pOLiCV 

Stanford UfixvarBity 

In this paper we will argue the advantages of the adversa^ 
evaluation modal. Our attempt is to persiiade the reader that this 
approach has substantial merits and that it ought to be used more 
,^ frequently than it is* A couple of caveats are in order before wa 
proceeds 

In one senea/ this syitiposium is an advergarial^ examination of 
the adversaxy model. We will discuss only the positive aspects of 
the adversa^ model. Dr. Poph^ and Dr# Carlson have the more 
difficult task of identifying its deficits , . 

Of course f in any adversarial endeavor it is . important to 
ensure a balance between the teMis, The Hawaii 3 on 2 evaluation 
included two trips to that island paradise. The fact that the 
co-*authors of this paper withdrew from tha evaluation prior to 
tha second trip to Hawaii is perhaps sufficient evidence of the 
relative wisdom of those arguing payoffs versus those arguing 
deficits* 



A paper presarited as part of a symposium^ The ^Adversary 
^ Evaluation Modelr A Second Look ^ Annual Meeting of the ^erican 
€^ Educational Research Association^ New York, April 4-^8? 1977* 
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Second, we base our arguments on a single eKpariance with the 
model— the evaluation of the Hawaii 3 on 2 program* The literature 
we reviewed was inforTnative , but essentially we are generalising 

from one case^-'always a dangerous praotice. Since that situation 

has obviously not hindered our colleagues/ we will attempt to be 
as persuasive in proselytising as they are in condemning* 

Peeling the Patina of Frofessionalism 
Many evaluations^ like novels ^ are written in the omniecient 
third person. The evaluator poses as an entirely objective 
exteamal party with no vested interest in that which is being 
evaluated* In truths we are, none of us, so removed^ so above it 
all, as to lack any bias* Perhaps unconsciously, our pre-dispositions 
prompt us to plan the evaluation in a certain way. Likely, these 
biases influence our choice of instruments or data collection 
techniques, as well* 

After an evaluation is concluded and reported, we might 
receive some gentle chiding from colleagues with a different 
prospective* Seldom, however , do these differences result in a 
contentious or public dispute. There are some exceptions, for 
example, "Sesame Street, Revisited", but these. ^ re joinders seldom 
receive any great notice, and interest soon dies out. 

The adversary approach not only pemits, it ensures such 
disputes, and it virtually guarantees that these arguments will 
occur early enough to have a positive impact on the evaluation * 
After a short prelude of congenial fellowship, the two teams can 
drop all vestiges of gentility and enter the fray with enthusiasm. 
There are two areas where this tension has real payoff*. 

■ -....3. ■ . ■ ' - ^ ' 



Planning the Evaliiation 

The 3 on 2 program^ like Topsyr just grew* There was no 
predeterinined plan for its rapid expansion * Some 3 on 2 classes 
used the Hawaii English Program (HEP) ? others did not* Soma self 
contained classes used HEP, but not, proportionately , as many as 
the 3 on 2 classes. Neither the 3 on 2 program nor HEP was equitably 
distributed across socio-economic levels. In the face of these^ 
and other confounding variables^ it was impossible to contrive an 
entirely satisfactory sampling plan. 

So we argued ^ almost interminably. The result waS/ in our 
view/ worth the effort. Each team wanted to ensure that its side 
was not placed at a disadvantage. Double aiid triple checking 
became the order of the day. Because of this attitude, we 
discovered and corrected certaiii misinformation about where the 
3 on 2 classes were located in time to revise the sampling plan. 
This new plan avoided a z^eal hazard In that it took into account 
the hitherto unknown disproportionate allocation of 3 on 2 across 
SEE levels. It wasn't perfect, but the fact that both teams could 
live' with the final sample plan tella us that it was probably as 
good as one could get. 

The whole process reminds one of the terse maxim of the New 
England carpenter "measure twice^ cut once." 

Choosing Instruments 

The perspectives of the two teams who evaluated the 3 on 2 
program concerning the measurement of student achievement were 
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quite different. This divergence came about, in the main# because 
of disagreements about fundamental measurement issues. 

The chairman of the advocate team was^ after alli* a leader 
of the objective referenced testing movement. 

Compromise in the choice of instruments was never a real 
possibility v The arbiters wisely agreed t© e^end the resources 
necessary to imdertake both nom referenced and criterion referenced 
testing* (The results were the same in both instances'-^no 
significant differences * ) ^ 

The evaluation of the 3 on 2 program occurred on an extremely 
tight schedule* The first team meetings did not occur until April 
of 1976* The final report had to be presented in January of 1977, 
It was not possible to administer or obtain a measure of the 
students' aptitudes prior to their entry into a 3 on 2 or a self^ 
contained class* After considerable debatei^ however^ we were able 
to agree that the administration of the Raven * s Progressive 
Matrices test at the same time as the norm referenced Spring tests 
would suffice as a covariate. 

The development of the questionnaires used in the 3 on 2 
evaluation is probably a better Illustration of the adversary 
model's payoff in this area* The developers of these instruments 
struggled to attain "balance" rather than "neutrality*" Provocativej 
one-sided questions were permissable as long as the instrument^ 
on balance^ was acceptable to both teams # to the project 

directors. Thus we were able to include in the teacher questionnaire 
Likert type items with stimulus statements such ass 



In the 3 on 2 Program 

c. Children do not receiva adeqiiate discipline* 

e. Children in a 3 on 2 class learn from each other, 

f* Children are distracted from learning by the noise and 

movement in the class* 
g. Older children waste time tutoring younger ones* 
E. The principals have not provided adequate support for 

the 3 on 2 teams* - 
Such questioni got right at the heart of the arguments 
dearest to proponents and opponents of the program/ and provided 
our respondents an opportunity to address themselves to issues 
which they knew to be real ^ not bland restatements. 

The response rates for our questionnaires were 76,5% for 
teachers, 82 ,3% for principals ^ and 50*8% for parents. No follow 
up procedures were used. This high return is partly accomited 
for by the controversial nature of the 3 on 2 program* We 
speculate, however, that it was also due to the fact that (by means 
of interviews and a pilot test of the first draft questionnaire) 
each team ferreted out the issues of concern to that portion of 
the public (pro or con) it represented, and saw to it that those 
questions got asked. 

Removing the Methodological Manacles 
Most evaluations seem to manifest an implicit adherence to 
Thorndike's belief that "If something exists, it exists in some 
quantity, and therefore, it can be measured," The aphorism may 
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in time prove true* As of now, however, we certainly do not 
possess adequate instruments for the investigation of anything 
approaching the broad spectrum of questions that ought to be 
addressed in a aomprehensive evaluation* We need to include many 
information gathering methods in order to even begin to rough 
in the sketch. Yet^ we must also be concerned about objectivity. 
There are too many uncorrabo rated/ unsubstantiated and unchallenged 
"observational studies" in our literature already. The adversary 

■ - . i . - ■ - ■ 

evaluation model has real payoff in that it proniotes the use of 

■-■ - , .... 

multiple types and sources of data and grants an investigative 
license to dig out the data to a greater extent than more 
traditional approaches, ! 

Multiple types and sources of Informiltion 

The adversary evaluation model is based upon a judicial system^ 
which relies on human testimony ^ witaess and expert* This 
testimony is subject to cross e^^amination and the corroboration or 
refutation of other witnesses and experts. The adversaty model 
thus explicitly encourages the use of ; non-^quantitative information. 
Quantitative data is most assuredly permissable ^ but it is not 
per se more valuable than other infomation. 

One cannot argue that the adversary evaluation model is 
imique in this regard. Many non-^adversarial evaluations have 
included a wide spectrum of information gathering mechanisms. 
The adversary model is distinctive^, however, in that, by its very 
nature, it fosters the use of multiple sources and types of data. 



^ The adversary evaluation model is not necessarily judicial,, 
of course. It can be structured as a debate or an arbitration; 
prQceeding. The nature of the evidence pe mi s sable is the same 
in^any case* ■ .rj::.:.: 



Since each side is seeking whatever information is most 
supportive of it^ position, a wider diversity of data may be 
collected in an advatsary evaluation* In the 3 on 2 evaluation 
the teams used the following approaches; 

m Norm referenced measiires of student achievement 

m Criterion referenced measures of student achieveinent 

m lnter\^iews with 
Legislators 

Members of the State Board of Education 

Staff of the Departntent of Education 

Principals 

Teachers 

Parents 

Students ' 
m Student attitude Inventories 

# Questionnaires 

Principals 

Teachers 

Parents 

m Analysis of Teacher Absenteaism Records 
m Analysis of Cost and Budget Information 

• Classroom Observation 

Structured 
Unstructured 

. • Analysis of Historical Docuinents 
. « Public Meetings 

This panoply of information was not part of our original plans 
rather it resulted from the sometimes almost desperate search for 
evidence favorable to one's cause^ Indeed It was: somewhat amusing 
to observe hard nosed testing specialists willingly praise the 
virtues of and press the necessity for classrodm observation and 
description. It was rather like watching Hemingway running waok^^ 
with a, butterfly net, 8 



Other evaluation methods use both quantitative and qualitative 
data, of course. The important difference between the adversary 
model and other approaches rests in the sphere of objectivity. 
Objectivity has been defined in terms of an agreement among 
competent observers* When the advocate and adversary agrees one 
can rest fairly certain that the phonomenon is real. 

Viewed from a different perspective/ meta-'evaluation is built 
into the adversary model. You don't dare crawl too far out on any 
evidential limb. The opposing side is observing your perch — saw 
at the ready . 

Digging out the Data 

The adversary evaluation model also pays off in that it grants 
evaluators investigative licenEe, They become detectives with a 
client. There is no need to bend over backwards to avoid offending 

Those who were interviewed in the 3 on 2 evaluation were 
usually interviewed simultaneously by one person from each team. 
These "witnesses" understood^ and frankly seemed to enjoy the 
situation. Tough questions were eKpected, The evaluators had the 
right/ indeed the obligation/ to ask for supporting details 
whenever a statement about -the program was made. We also had the 
right to search for evidence that contradicted the testimony of 
even the most . influential * 

Some examples of what this investigative license can lead to 
are in order. We interviewed district superintendents/ principals 
and teachers about such issues as how the decision was made to 
install the 3 on 2 program in a given school. The answers were 



often contradictory. The principal would state he left the decision 
up to his teachers, for escampler but the teachers "Would insist that 
they had no voice in the matter* Enough confusion was evidenced 
in our preliminary Interviews for us to address this and similar 
issues in the questionnaires. As a consequence we learned that s 
considerable confusion existed about who was responsible for 
decisions about implementing the 3 on 2 program. This evidence on 
the lack of clear leadership might not have been as vigorously 
pursued or as frankly reported had another evaluation approach been 
used, 

A more pointed example of this payoff occurred in the examination 
of the costs of the 3 on 2 program. We were originally lad to 
believe that the cost of the program was about 6 million dollars a 
year. Indeed that figure had been widely circulated and accepted 
as accurate by a number of those with "whom we talked. By some 
calculations the 6 million dollar figure seemed perfectly accurate. 
Cost was a major issue in this evaluation^ however^ and both teams 
wanted to leam as much as they could about the sKpenditures 
required for the 3 on 2 program. Naturally the advocate team 
hoped to find lower costs or cost savings; whereas the adversary 
team wanted evidence that the program was inordinately expensive. 
So by digging into various budget reports and asking probing 
questions of several individuals , the two evaluators (one from 
each team) assigned to cost data determined that the program 
actually cost in excess of 9 million dollars per year. Without 
the legitimacy of a cause to represent^ we question whether they 
would have been as persistent or as insistent in securing accurate 
infomation. 



Pleasing The Paople 

Evaluation im often perceived more as beast than beauty by 
those who call upon its services Its unpopularity stems from 
many factors ^ but perhaps primary among them is the insensitlvity 
with which traditional evaluation approachas are often applied. 
Any evaluation has many audiences # each of which has its own 
questions of interest. Only a small portion of these queetions 
usually get asked* Moreover, most evaluations ignore the vast 

.. .. ' ' ..... 

bulk of those who want an opportunity to express their opinion or 
who at least want to be informed about the progress and results 
of the evaluation. 

The adversary approach promotes the use of multiple types and 
sources of data. It also has a distinctive and public means of 
reporting* The 3 on 2 program featured, for eKample, an hour long 
televised presentation of a debate between representatives of each 
team. Because the people affected by the decisions to be made 
have more of an opportunity to be involved in the decision making 
process, we believe the adversary approach elicits their support 
rather than their enmity. To check this perception we decided to 
use a simple questionnaire\i 

The Questionnaire Study \ \ 

The questionnaire which we used was developed jointly by 
Blaine R, Worthen and William J. Wright. It was intended for 
those most immediately involved in making or implementing 
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decisions about the, future of the 3 on 2 prograiri* It concernad 
only the 'Value and utility of the adversary approach. A copy of 
that questionnaire with the number of individuals who responded 
to each response option filled in is appended to this paper. 

The questionnaire was sent to 34 individuals including each 
member of the Hawaii Board of Education/ each district and deputy 
district superintendent, and the principal staff ineiffcers of the 
Department of Education. 

One of our respondents was too ill to oblige us. Of the 
remaining 33/ 27 or 82 percent/ returned the questionnaire.f some 
anonymously. We will report on those items most directly related 
to client satisfaction in this paper* Blaine Worthen will address 
the remaining issues in his presentation. 

The majority of. those responding to the questionnaire (78 
percent) believed that "the advocate^adversary approach more clear 
focuses on jLssues relevant to the decisions to be made." Only one 
respondent felt that "the advocate^adversary approach tends to 
confuse rather than illiaminate the issues." 

Among the. other characteristics identified by our respondents 
as descriptive of the adversary evaluation approach were the ^ 
following: ... . 



Since the directions stated "check any that apply^" and 
responses were not polar/ the sum of those who chose a particular 
set of statements v/ill not equal 100% except by chance. 



• V "The advocate-adversmry approach provides . more cQmprehens ivs" 

information about the program being evaluated* " (55 percent) 

• "The advocate-adversary approach results in a more balanced 
presentation—important issues get overlooked." 

(78 percent) 

Few of our respondents fait that the following statements were 

true:.. ■ . " 

-■■-m "The advocate-adversary approach depends too much on the 
personality characteristics of the spokespersons for the 
two teams." (33 percent) 
V "The advocate-adversary approach focuses too much attention 

on the evaluation method rather than on the program being - 
; evaluated. " (f^^percent) : / 

® "The advocate-adversary approach does not provide decision 
makers with the evidence they need to make a choice." 
(7 percent) 

The most imprassive evidance of tha payoff of-the adversary 
approach with respect to client satisfaction came in response to 
the question "Would you recommend the use of the advocata-adversary 
evaluation approach to others?" Of the-25 individuals who answered 
tha question/ 10 (or 40 percent) responded "definitely yes"; 14 
(or 56 percent) responded "probably yes"; 1 raspondant was 
"uncertain". There were no negative responses to this quastioni 

^ Conclusion ~ " ^' 

In this saction of tha paper we will review briefly the 
payoffs of the adversary modal. We will then attempt to look 



beyond the 3 on 2 experience and suggest some variations to the 
adversary modal that might ba intaresting to try out. Finally> 
we will look at what may be the biggest payoff of all. 

Sunmlng up the Payoffe _ , 

The payoffa of the adversary approach are many. The listing 

which follows is intended as a quick surm 

.m Planning * • 

The adversary model has real benefits in that^ as the 
two tsMs plan the evaluations debate about critical 
issues such as problem definition.> sampling plans 
and the like occur. Thus the critical input of 
colleagues is considered at Uie most opportima 
momerit-^ prior to actual investigation. 

m ■ Ins trumentation , _ 

. . ■ . . f - ..... 

Because of the adversarial nature of this approach^ 
the instriiments selected or developed are li 
be of higher quality. In the development of 
questionnaires"and interview schedules tha focus is - - 
on balance not blandness. The inclusion of 
provocative questions which address the real issues 
of various sub groups is thus encouraged. 
m Multiple Types and Sources of Information 

The adversary model axplicitly encourages the use of 
" : qualitative- as wall as quantitative inforinationr lt^ ^ 

compels the evaluators to search vigorously for any 
credible evidence supportiva of the position they 
, champion. Yat, because of the constant presence of : . 



^valuators who represent the cqntraa^ positi©n> 
objectivity # in the sense of agreement among 
competent observers/ is at a premium- When 
advocate and adversary report observing the same 
- phenom^ can have oonf idence in: its reality. 

Investigative License 

The advocate or adversary evaluator represents a 
particular cause. In this situation he or she has 
more freedom to challenge testimony and to dig into 
records^ reports^ Md other sources "of infozi^tion 
seeking corroborative or contradictory evidence. 
Sometimes^ when you seek to ferret out the truth, 
you find it. 

Client Satisfaction^ ^ \ . v - 

Because the advocate and adversary e valuators 
represent the polar positions within some larger . 
community /-people feel that the is believe 
to be important are not overlooked. The public 
nature of the reporting mechanism also involves 
V people . They don ' t feel as excluded from the 
decision making process. In a study conducted 
among our principal client group/ all but one 
respondent to a question concerning whether they 
_ would reconunend the-- adversary model to others - ~ i 
reported that he or she would probably or definitely 

. ^ . re c ommen d the . adve r s ary . e va 1 ua t i on mo de 1 , t o ^ o the r s , ^ ^ 



Built .in Meta EvaluatiQn 

Running through all the payoffs is the notion of a 
built in ma ta evaluation with a distinct advantage* 
Instead of a carping critic of the Monday inorning = 
quarterback variety^ the meta evaluator is an 
opponent, who is there whan needed, ^d who has as 
large an interest in the ultimate credibility of the 
evaluation as you do. ■ : . 



Looking Bayond the 3 on 2 Experience 

The approach used in connection with the 3 on 2 program is . 
but one type of adversary evaluation model, Owens and Hiscox 
have described other versions in their paper/ We would like to 
-suggest that the invastigation of other judicial models may provide 
some inLoresting variations. In some European courtrooms, for 
example, the judge is not an impartial monitor, but is charged 
with the rasponsibility of seeking the truth of the matter. The 
judge is free to, and often does, cross question witnesses as 
vigorously as either of the attorneys. In certain systems all 
merrtDers. of_the bar„have an equal responsibility to find the truth. 
The def ansa, prosacution, and judge work toward the same end, but 
from differant perspectives. Facts are established and clearly 
distinguished from opinions and racollections which are disputable. 
Perhaps something akin to this approach might be of value in 
education, , . . . .. ...... 

Yat wa must be caraful to laarn what we can from our own 



judicial system before we move on too quickly to other models. 
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Consider .the . fact that those who pass our lawsr those who en force : 
the laws r those who dispute the legal issues of the day and those 
who sit in judgement of: these disputes all tend to be meinbars of 
the sama professional society /Considar the benefits of such a \i 
system. There Is much to be learned indeed • 'y..:':... .Z.^^].. -^i ^ 

The Big Payoff 

The big payoff of : the adversary evaluation modal Is that it 
forces us ^ as a profassion^ to explore alternative means of 
obtaining and reporting information. It is not tha answer # but 
it is one mora arrow in our arsenal^ and w^* need all the weapons 
we can get* „ 
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•Now thac you have had an opportunity to read the written reports and hear 
the oral reports of the 3 on 2 evaluation/ do you feel that the advocate-^ 
adversary approach produces info nnation worth the cost of having two 
teams involved? (check ONE) 

The Inforaation producsd by advocate-adve^^ was 
worth more than the e^mluatlon's cost r ; , ■ 

The infonnatlon produced by the advocate^adversaiy evaluation :/ 
was worth the evaluation's cost 

The Information produced by the advocate^advefsary evaluation " ; ; 

was not worth the evaluation's cost 

Compared to other methods'of evaluation, which of the following a tatei^enta 
about the advocate^adversary approach to evaluation do you think are true ? 
(Check ANY that apply) 

The advocate-adversary approach more clearly focuses on issues 
relevant to the decisions that are to be made 



The advocate^adversary approach tends to confuse rather than 
illuminate the Issues 

The advocate^adversaiy approach depends too much on the 
personality charaLCteristics of the spokespers the two 

-teams ; ■^ ■ , 

The advocate-adversary approach.prQvides more comprehensive 
Information about the pfogTam^^b^^ 

The advocate-^adversa^ approach results In a more balanced 
presentation—important sub-issues do not get overlooked 

The advocate^adversan^ approach focuses too much attention 
on the evaluation method rather than on the program being 
evaluated . \ l ; 

The advocate^adversary approach presents more nlearly defined 
decision alternatives and the ' 

The advocate-adversary approach does not provide decision 
.makersJwith the evidence they need to make a choice 



In which" of the following circumstances do you feel it 'would be appropriate 
to use an advocate-adversary evaluation? (check ANY that apply) 



When decisions must be made about whether to continue or terminate 
a program ■ : ^ - 

When decisions must be made about how to improve a program 

When the evaluatton Is condu by external evaluators (persons 
outside of the program being evaluated) - 

When the evaluation is conducted by Internal evaluators (parsons 
employed by the program being evaluated) 

When the program being evaluated Is large, and affecta a large _ 
number of people 

Whan the program being evaluated is controversial and many people 
hold strong positive and negative opinions of it , 

When there are many different audiences for the evaluation report 

When it is taportant to have an interestiug non-written method of 
reporting the evaluation results 

Other (please explain) ^ 



Would you recommend the use of the advocate-adversary evaluation 
approach to others? !. 



Definitely Yes Probably Yes Uncertain Probably No Definitely No 
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37% 



S2% 



4% 



What do you feel are the unique advantages (if any) of the advocate--^ 
adversary approach to evaluation? 



What do you feel are the unique disadvantages (if any) of the advocate- 
adversary approach to evaluation?. 



